System for processing video signals

ABSTRACT

The invention relates to a system for processing video signals, the system comprising a receiver ( 100 ) arranged to receive video signals, at least one video frame of which comprises at least one area corresponding to a respective one of a plurality of broadcast data sources. The system comprises a processor ( 150 ) arranged to process said video frames to extract at least a part of said area from at least one of said video frames. The frame ( 200 ) of a mosaic video signal may comprise one or more small pictures ( 210 ) and each of them may render a miniature version of a television program. A detection of edges and a detection of lines can be performed for extracting said area ( 210 ) or sub-area ( 240 ) from the frame. The television channel, from which the extracted area or sub-area was received or with which it is associated, can be identified. The user is enabled to specify a representation of the extracted area or sub-area on a display screen.

The invention relates to a system for processing video signals, thesystem comprising a receiver arranged to receive video signals, at leastone video frame of which comprises at least one area corresponding to arespective one of a plurality of broadcast data sources.

U.S. Pat. No. 5,633,683 discloses a television receiver for receivingand displaying a mosaic video signal including a plurality ofsub-pictures, each sub-picture representing one of a plurality of videosignals, and position data linking a position of each sub-picture on adisplay screen with a program number of the associated video signal. Thereceiver comprises a decoder for decoding the program number of a videosignal represented by the position of the sub-picture being displayedand pointed at by a user. Thus, the user can select a desired programfor watching, recording etc by simply “pointing and clicking” thecorresponding sub-picture on the mosaic screen. The receiver furthercomprises means for deriving the selected sub-picture from the mosaicsignal and means for simultaneously displaying the selected sub-pictureand the associated video signal for a predetermined period of time afterselecting said video signal. In that way, the receiver comprises a firstdecoder for decoding and displaying the selected program and a furtherdecoder for processing the mosaic video signal. The sub-picture beingcharacteristic of the selected program is cut out and displayed as apicture-in-picture.

The sub-pictures can be derived only when the position data is obtainedby the television receiver. Such an arrangement requires the specialtransmitter adapted to generate the position data, which is a limitationof using such a technique. For example, if the equipment of a televisionservice provider cannot be arranged to generate the position data, thesub-pictures cannot be derived by the receiver.

It is an object of the invention to provide a system of the kind definedin the opening paragraph which can suitably operate with the videosignal regardless of obtaining said position data linking the positionof each sub-picture on the display screen with the program number of theassociated video signal.

The object of the invention is realized in that the system comprises aprocessor being arranged to process said video frames to extract atleast a part of said area from at least one of said video frames.

The frames received by the receiver comprise one or more areascorresponding to the respective broadcast source. For example, the framecomprises small pictures and each of them may render a miniature versionof a television program broadcast on the respective television channel.Usually, the pictures have a rectangular form. The receiver comprisesthe processor arranged to graphically process video data of each frameand extract the complete picture from the frame. In another example,video information broadcast on one television channel occupies a wholeor part of the area of the frame, and only part of said videoinformation, sub-area or sub-picture is extracted by the processor. Saidsub-area, e.g. a weather map or graph figure, may have the rectangular,circle, or any other shape forming a closed figure.

First, a detection of edges can be performed for retrieving said area orsub-area from the frame by the processor. Thus, edges between areas oredges between a sub-area and other video data are determined. In one ofthe embodiments, the detection of edges is performed on a plurality ofsequential frames, e.g. by using a temporal filter, where a position ofthe area or sub-area is the same on these frames. The detection of edgesis more reliable when the processor analyses more than one frame todetermine edges of the area on the frame. Therefore, the edges betweenareas within the frame or the edges between sub-areas within the areaare made stronger than other edges which exist in the frame and changeover time, e.g. due to the motion of objects, characters, in it. Thedetection can be improved by further performing a detection of linesdelimiting the area or sub-area from the other video data, e.g. a Houghtransform can be used for such a detection. In the case of the mosaicframe, a property that lines between pictures are equidistant may beused during the extraction process to reduce or avoid possible spuriousdetection. When horizontal and vertical lines for the correspondent areaor sub-area are determined, the video data of that area or sub-area maybe further processed and/or transferred to a presentation means such asa display device for showing the extracted information. It is anadvantage of the system according to the present invention that no dataindicating the position of areas in the frames are needed. This gives apossibility to extract the video information independently of whatauxiliary data for processing the information are available, withoutunnecessary restrictions given in the prior art.

Alternatively, the receiver comprises a marker with which a user canindicate the area or sub-area of the frame shown by the presentationmeans, which is to be extracted from at least one of the video frames.The processor need not determine said area or sub-area by using thedetection of edges and lines because the area or sub-area to beextracted is readily specified by the user. In other words, the user maybe enabled to mark some part of the frame or frames shown on a screen,and that area or sub-area will be extracted from one or more frames. Itis another advantage that the present invention provides more ways ofprocessing the video information than is known in the prior art.

In one of the embodiments, the receiver further comprises means foridentifying the broadcast data source, or a television channel, fromwhich the extracted area or sub-area was received or with which it isassociated. When the area or sub-area is extracted, it may be presentedas video data without any knowledge of what happens with these datanext. If the area or sub-area were extracted from the mosaic videosignal, it might be unknown to which channel they correspond. There aremany ways of identifying the respective channel. For example, a logo ofthe TV channel may be present in the extracted area or area from whichthe sub-area was extracted. The user may also specify the channelmanually.

In another embodiment, the presentation means may enable the user tospecify a representation of the extracted area or sub-area on thescreen. For example, a position of the extracted information may befitted to the user's preferences, the screen area in which the extractedarea or sub-area is shown may be modified, etc.

These and other aspects of the invention will be further elucidated anddescribed with reference to the accompanying drawings, wherein:

FIG. 1 shows a functional block diagram of the receiver of the systemsuitable for implementing the present invention;

FIG. 2 shows a visual presentation according to an embodiment of thepresent invention, in which the mosaic screen is shown;

FIG. 3 shows an example of a visual presentation according to anembodiment of the present invention, in which the sub-area extractedfrom the mosaic screen is shown in a subsidiary screen area, whereas theareas extracted from the mosaic screen are shown in other subsidiaryscreen areas;

FIG. 4 shows an example of extracting the sub-area corresponding to onebroadcast data source to be shown simultaneously with the framescorresponding to another broadcast data source.

FIG. 1 is a functional block diagram of a receiver according to thepresent invention. The system comprises a receiver 100 which may beconnected to or comprise a display device (not shown), and VCR (VideoCassette Recorder), loudspeakers or other devices. The receiver may alsobe integrated into different devices such as set-top boxes or otherdevices designed for operating with AV (audio-video) signals. Thereceiver 100 receives a plurality of video signals transmitted via asatellite, terrestrial, cable or other link. A command can be inputtedto the receiver by an infrared signal transmitted from a remote controlunit (not shown). Thus, the receiver comprises a receiver (not shown)operating with control signals. The remote control unit may have specialbuttons associated with possible commands for controlling the receiveras it is described herein. Nowadays, MPEG-based systems for transmittingand/or receiving digital video signals are well known. The receiveraccording to the present invention may be arranged to receive digitaland/or analog video signals.

The receiver comprises at least one tuner 110, a demultiplexer 120, anoptional audio decoder 130, at least one video decoder 140 and a videoprocessor 150. The received video signals are applied to the tuner 110.

The video signals may incorporate a mosaic video signal comprisingframes with small-size pictures occupying a relatively small area of therespective frame. Each picture represents video signals associated withthe respective TV channel, Internet broadcasting center or otherbroadcast data source. Alternatively, the video signals may compriseinformation received from a single broadcast data source. For example,the tuner may receive video signals of only one TV channel.

The tuner 110 may include demodulation circuits for demodulation of thereceived signal and error correction circuits for detecting andcorrecting any occurred error. The output of the tuner is supplied tothe demultiplexer 120 for deciphering the signal. The demultiplexerprovides the output audio signal to the audio decoder 130 and outputvideo signal to the video decoder 140. The decoders 130 and 140 decodethe audio and video signal, respectively, which may be a MPEG-compressedsignal. With the future development of video systems, the implementationof the present invention may be varied by the person skilled in the art.

The receiver may comprise more than one tuner and more than one decoder,for example two tuners and two video decoders. Each decoder may comprisea memory (not shown) for storing the video signal. One of the tuners maybe used to receive signals of a user-selected channel, while anothertuner is used to receive the video mosaic signals. In that way, one ofthe video decoders may be used for decoding the video mosaic signals.The signals corresponding to the different TV channels are received withthe mosaic video signal. It is an advantage of the present inventionthat, alternatively, the receiver may comprise only one tuner andperform the same functions as when two tuners are available. Forexample, the tuner may be arranged to receive the signals of the channelselected by the user, and receive the signals of the mosaic. The singletuner may tune to e.g. the mosaic channel for x, e.g. 3, pictures and tothe main, e.g. user-selected, program for 50-x pictures. The x pictureperiod leaves the tuner enough time to extract at least one picture fromthe mosaic channel. The missing pictures for the main channel can becreated by picture repetition, or in a high-end system with a naturalmotion interpolation. Because only a few pictures are missing from themain program, this is hardly visible to a viewer.

The decoder 140 supplies the decoded video signal to the video processor150. The processor processes the received signal according to thepresent invention to extract video information.

When the mosaic video signal is found to be available, the frames ofsaid signal are simple pictures, or video information, which areanalyzed without any additional data. FIG. 2 shows an example of a frame200 of the mosaic video signal. To identify areas 210, coordinates XY ofthe corresponding area, usually having a rectangular or rectangular-likeform, on the frame have to be determined. This can be done first byusing a detection of edge technique generally known from the book“Two-dimensional signal and image processing”, Jae S. Lim, Prentice-HallPTR, New Jersey, 1990, pp. 476-483. The edges between areas 210corresponding to different TV channels have to be detected. The edge isa boundary or contour at which changes in physical aspects of the imageoccur, for example changes in pixel gray value, color and texture. Wherethe physical aspects change rapidly, an edge line from a strip ofcandidate edge points is determined according to the algorithm describedin the above reference. For example, a threshold value for the physicalparameter can be determined by computing the gradient of vectors in thex- and y-direction. A magnitude of the gradient is then compared withthe threshold to determine candidate edge points. After that the edgeswill appear as strips, and e.g. an edge thinning algorithm is applied todetermine an edge curve. In one simple edge thinning algorithm, the edgepoints are selected by checking if a module of the gradient is a localmaximum in at least one direction. The detection of edges may beperformed on the sequential frames for the same area to ensure thecorrect detection.

A detection of lines 220 is performed by the processor 150 after thedetection of edges. Many methods for finding an alignment of points inthe image and arrangement of features are known. For example, aso-called “least features” method where a sum of squares of verticaldeviations of each point from the line is minimized can be used to fit astraight line to data points. A superior Hough method may advantageouslybe used as it is known from “The image processing handbook”, John C.Russ, CRC Press, Boca Raton, Fla., 1995, pp. 495-500. It should be notedthat said method can also be used for detecting areas bordered bynon-straight lines in the frame. When a fit of a line to the detectededges is determined, the necessary video area is known to the system. Ifthe area has the rectangular form, coordinates XY of the area on theframe are determined, while the video data corresponding to the area canbe identified and separated from video data of the complete frame.

When the area 210 corresponding to the respective broadcast data source,or TV channel, is extracted, there may still remain an uncertainty towhich TV channel the area corresponds, or it may be simply unknown tothe system from the received frames, or simple images. For this purpose,the processor 150 may be further arranged to identify the source of theextracted video area. One of the possibilities is that the processoranalyzes the entire area 210 for locating a logo 230 of the TV channel,if it is available. Then, a recognition technique known in the prior artis applied to recognize a sign, logo-image, text, etc. which is thencompared with identification information of the channels, previouslydetermined by the system or predetermined by the manufacturer, stored inthe receiver.

Alternatively, in a phase of determining the location of a logo, theuser may “manually” highlight the logo video area on the screen. Atemporal analysis may be applied for the logo video data, e.g. bydetecting it in the sequential frames, over the number of frames. Thelogo data can be identified by using the detection technique describedabove, extracted and stored in the memory coupled to the processor. Logoidentification data, a logo template, obtained in such a way may be usedfor identifying the TV channel. To identify the channel, the logo in thevideo signals may be correlated, e.g. using the well-known least squaresmethod, with the logo template stored in the memory.

It may happen that the logo is not present in the frames when acommercial occurs. However, positioning of the area with informationbroadcast on the specific TV channel usually does not change with everynext frame. This problem may be resolved by storing, in a memory means(not shown) coupled to the processor, a table of the TV channelidentificators associated with the respective areas, as it is shown forthe mosaic frame of FIG. 2 in Table 1. This table may be derived once bythe system itself and stored. TABLE 1 SBS-6 Yorin Ned1 CNN BBC RTL4 V8RTL5 Ned2

Alternatively, audio information corresponding to the image the TVchannel of which is being established may be analyzed. Some TV channelsperiodically repeat broadcasting their own promotional pause, andaccompany video information with a specific music. All of this may alsobe used for identifying the TV channel and implemented by the skilledperson without difficulties.

In a further example, data for identification of the broadcast sourcesmay be received from the remote transmitter. For example, a sequence ofchannel names (ID) or abbreviations received by the receiver 100 issufficient for this purpose. Said data may comprise one specialcharacter (SC) specifying that a right, or other direction should beused for associating the identifier with the next area. In this way,only little additional information is needed. An example of suchidentification data for the mosaic frame of FIG. 2 is shown in thefollowing Table 2. Implementation issues of such an identificationmethod can be found in U.S. Pat. No. 5,633,683. TABLE 2 SC ID

SBS-6 Yorin Ned1 CNN BBC RTL4 V8 RTL5 Ned2

According to a further aspect of the present invention, the processor150 is arranged to extract only part of the area 240, or sub-area,corresponding to the respective broadcast data source, or TV channel.Such a sub-area may be extracted when position data for identificationof the sub-areas in the area are generated by a transmitter, orbroadcast data source, to the receiver and used by the processor. Forexample, the TV channel broadcaster may include such data in the digitalvideo signal. A provider of mosaic video signals may incorporate thatdata in the mosaic signals for areas corresponding to the respective TVchannel, if available. Such a system comprising the transmitter andreceiver may be realized by the appropriate modification of the systemknown from U.S. Pat. No. 5,633,683.

Alternatively, the user can specify the sub-area to be extracted. Forexample, a cursor, marker or other pointer indicated on the frame 200,which is shown on the display device, can be used for such purposes. Theuser can use direction keys of the remote control unit to displace themarker, a cursor, and special keys for marking the sub-area 240 on theframe. When the sub-area is user-operably selected, the sub-area can beextracted from the next frames of the mosaic video signal.

It should be noted that if the user manually selects the wholerectangular area 210, the selected area corresponds to the rectangulararea automatically detected, using the described technique, for examplethe Hough transform.

Another way of extracting sub-areas may be related to MPEG-4 standardproviding possibilities of manipulation with video objects. Other waysmay be derived within the scope of the present invention. Theidentification, if necessary, of the TV channel corresponding to theextracted sub-area may be performed as is disclosed above.

FIG. 4 shows a frame 400 of video signals of the TV channel. In contrastto examples stated above, all information in this frame corresponds tothe same channel “A”. The sub-area 410 can be extracted in one of themanners disclosed with reference to the mosaic frame.

An identification of objects, such as people, an animal, a character,car, etc, within the areas may be performed by using the techniquedescribed above. The user may choose one or more of the identifiedobjects to be further identified in the subsequent frames. Of course, ifthe object is a three-dimensional graphical figure, a presentation ofthe object may change over time. Therefore, an analysis of presence ofthe same object in different frames can be implemented. For example, inan MPEG-4 standard, a control of a video object is known, said videoobject may be matched with the objects shown in the next frames. Theimage of selected object extracted from the frame can be furtherdisplayed or processed. A set of the extracted object imagescorresponding to the specified object may also be stored for furtherdisplay, or the like.

Generally, two types of extraction according to the present inventioncan be summarized. In the first case, the area or sub-area is extractedwith respect to the position and size of that area or sub-area on theframe and independently of what is the content of that area or sub-area.In another case, the processor extracts the objects provided in the areaand this object may be extracted from any part of the area wherein it isphysically present.

It should be noted that the mosaic frames need not necessarily bereceived from broadcast TV signals. The receiver may comprisecommunication means for receiving, and may also be transmitting, digitalvideo and audio content from the Internet.

The extracted video information can be further used for displaying it inmany manners. The video processor may comprise a picture-in-pictureprocessor (P-in-P) (not shown) or video switch functioning as it isknown from U.S. Pat. No. 5,633,683, respectively. The extracted videoinformation is further applied to the video switch. The video processor150 can be suitably programmed to perform all functions disclosedherein. The audio and video output of the receiver 100 is furtherprovided to AV devices for rendering audio and/or video content.

FIG. 3 shows an example of frame 300 presenting the extractedinformation. On frame 300, a main screen 310 wherein a program selectedby the user or in other ways is shown. Subsidiary screens 320 aresmaller than the main screen 310, for example, because the user does notlike TV programs shown in the subsidiary screens so much as the programin the main screen. Video content shown in the subsidiary screensprovides the user with information being broadcast on different TVchannels and can be extracted as it is disclosed above, for instance,from the mosaic video signals. A frequency of refreshing the informationin the subsidiary screens should be sufficient to keep the usersufficiently informed, for example, one frame of the respective TVchannel per second or one frame every half second is shown. Thefrequency depends greatly on many factors such as number of tuners 110available in the receiver 100, processing power of the processor 150,type of information provided to the receiver etc. It is also possiblethat “live”, real-time, video content is shown in the main andsubsidiary screens.

The extracted sub-area may be shown in the same subsidiary screens asthe extracted areas. Both extracted areas and sub-areas may be scaled,or mapped, to an area on the display screen predetermined by the user orsystem. As an example, a CNN “ticker tape” with “moving text” of news isextracted from the mosaic 240 and shown in the subsidiary screen 330.

The user may be enabled to change an arrangement of the subsidiaryscreens 320 in frame 300. A position of the subsidiary screen may bechanged and the screen can be moved within the frame area. For example,the user would like to have the screen 330 at the upper side of theframe 300. For this purpose, the remote control unit may have a specialbutton for switching the TV set to an editor mode. A display menu withcommands for creating a new subsidiary screen, deleting, moving,changing a size, etc. of the subsidiary and/or main screen may be usedin the editor mode. The user may also indicate the source of informationto be shown in the respective screen 310, 320 or 330, the frequency ofits renewal or other parameters. The user may be enabled to select anarrangement in which the extracted areas or sub-areas are scrolledsequentially through the subsidiary screen areas. A border, rim of thesubsidiary area can be made bold, highlighted, etc. The results ofediting the arrangement of frame 300 are further transferred to thevideo processor for controlling the extraction and adequate presentationof video content.

Another example of displaying extracted information is shown in FIG. 4.The sub-area 410 is selected for extraction from frame 400 of the TVchannel “A”. The sub-area 410 is shown on frame 450 and scaled to thesubsidiary area 460 having an arrangement with a black border. The imagein screen 460 can be shown in scaled or semi-transparent form anddistinguished by the black border from another TV program of channel “B”shown on the rest of the screen. Thus, the user may watch the contentfrom two channels A and B.

In one of the embodiments, a data retrieving system comprises thereceiver according to the present invention. Some of the areas orsub-areas may be stored in the memory with descriptions of theircontent. For example, the “ticker tape” 240 may be identified in thesystem with some descriptors such as “CNN news headlines”, “newsbanner”, etc. Generally, it can be done for TV programs in which suchsub-areas are permanently present. The position of such sub-areas in theframes can be stored together with said descriptions. The user may alsogive pseudo-names to the sub-areas. A search interface for retrieving adesired sub-area on the display may be enabled upon the user's request.Thus, whenever the user would like to watch such a sub-area on thescreen, he/she may easily retrieve it.

Other implementations, which provide similar functions, could besubstituted for the aforementioned implementations without departingfrom the scope of the present invention. The various program productsmay implement the functions of the system and method of the presentinvention and may be combined in several ways with the hardware orlocated in different devices. Variations and modifications of thedescribed embodiment are possible within the scope of the inventiveconcept.

1. A system for processing video signals, the system comprising areceiver arranged to receive video signals, at least one video frame ofwhich comprises at least one area corresponding to a respective one of aplurality of broadcast data sources, characterized in that the systemcomprises a processor being arranged to process said video frames toextract at least a part of said area from at least one of said videoframes, by using an image analysis algorithm.
 2. The system of claim 1,wherein said area or part of said area has a rectangular form.
 3. Thesystem of claim 1, wherein the processor is arranged to perform adetection of edges on said area or part of said area.
 4. The system ofclaim 3, wherein the processor is further arranged to perform adetection of lines delimiting said area or part of said area from therespective frame.
 5. The system of claim 3, wherein the processor isfurther arranged to perform the detection of edges on a plurality ofsequential frames for the area or part of the area corresponding to thesame one of data sources.
 6. The system of claim 1, further comprising amarker for user-operably indicating said area or part of said area to beextracted from at least one of said video frames.
 7. The system of claim1, further comprising identification means for identifying acorrespondence of the extracted area or part of the area to one of theplurality of broadcast data sources.
 8. The system of claim 1, furthercomprising a presentation means being arranged to show at least oneextracted area or part of the area mapped to a respective screen area.9. The system of claim 1, wherein said system is operable to switchbetween reception of the video frames comprising at least one areacorresponding to the respective one of the plurality of broadcast datasources and reception of video frames from a selected one of thebroadcast data sources, the presentation means being arranged to show,on a main screen area, said received video frames of the selectedbroadcast data source and, on at least one of a plurality of subsidiaryscreen areas, at least one extracted area or part of the area.
 10. Thesystem of claim 1, comprising a further receiver being arranged toreceive video frames of video signals from a selected one of thebroadcast data sources, the presentation means being arranged topresent, on a main screen area, said video frames of the selectedbroadcast data source received by the further receiver and, on at leastone of a plurality of subsidiary screen areas, at least one extractedarea or part of the area.
 11. The system of claim 9, wherein saidpresentation means are arranged to user-operably specify arepresentation of at least one extracted area or part of area the in therespective one of the plurality of subsidiary screen areas.
 12. Thesystem of claim 11, wherein said presentation means are arranged toenable the user to specify a size and/or a position of at least onesubsidiary screen area.
 13. A receiver arranged to receive videosignals, at least one video frame of which comprises at least one areacorresponding to a respective one of a plurality of broadcast datasources, characterized in that the receiver comprises a processor beingarranged to process said video frames to extract at least a part of saidarea from at least one of said video frames.
 14. A computer programproduct enabling a programmable device, when executing said computerprogram product, to function as the system as defined in claim 1.